<html>
<head><meta charset="utf-8"><title>rollups vs testing · t-compiler · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/index.html">t-compiler</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/rollups.20vs.20testing.html">rollups vs testing</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="204209957"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/rollups%20vs%20testing/near/204209957" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/rollups.20vs.20testing.html#204209957">(Jul 17 2020 at 14:08)</a>:</h4>
<p>So <span class="user-mention" data-user-id="116266">@Santiago Pastorino</span> and I were looking at this article <a href="https://hacks.mozilla.org/2020/07/testing-firefox-more-efficiently-with-machine-learning/">Testing Firefox more efficiently with machine learning</a>. The basic idea is to use some automated heuristics to learn which tests might be broken by which commits. </p>
<p>Mozilla's process is more "optimistic" than ours -- they land commits and then test afterwards, and remove things that break the build. The goal with the machine learning then is to figure out (after the fact, I think) what is the best order to run the tests to find problems efficiently and back them out.</p>
<p>Reading the article got me thinking about how rollups interact with bisection. i.e., if we were able to run a "pretty good" set of tests based on some heuristics and land each PR more efficiently, then we'd sometimes find that tests created unexpected failures. However, because we wouldn't have any rollups, we'd also be able to bisect and find the PR quite efficiently (and hopefully then back it out with minimal disruption). </p>
<p>Not really proposing a change, just thinking out loud. I'm not convinced that the "pessimistic" approach we're using now of testing everything up front is the best solution, though I think it has a lot of advantages.</p>



<a name="204209999"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/rollups%20vs%20testing/near/204209999" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/rollups.20vs.20testing.html#204209999">(Jul 17 2020 at 14:08)</a>:</h4>
<p>(And of course even our current pessimistic approach can't prevent all errors, i.e., we still get regressions in the wild)</p>



<a name="204210292"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/rollups%20vs%20testing/near/204210292" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/rollups.20vs.20testing.html#204210292">(Jul 17 2020 at 14:11)</a>:</h4>
<p>IMO for bisection enhancement we should modify bors to basically bors try any r+'d PR such that we have artifacts for it</p>



<a name="204210367"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/rollups%20vs%20testing/near/204210367" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/rollups.20vs.20testing.html#204210367">(Jul 17 2020 at 14:12)</a>:</h4>
<p>(alternatively we can setup a bot that'll "generate" such a set of try builds from a rollup post-facto, when needed -- it would be a bit of work but not too hard)</p>



<a name="204210421"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/rollups%20vs%20testing/near/204210421" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/rollups.20vs.20testing.html#204210421">(Jul 17 2020 at 14:12)</a>:</h4>
<p>yes, I've thought about that too</p>



<a name="204210433"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/rollups%20vs%20testing/near/204210433" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/rollups.20vs.20testing.html#204210433">(Jul 17 2020 at 14:12)</a>:</h4>
<p>this is also very similar to what FF does</p>



<a name="204210441"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/rollups%20vs%20testing/near/204210441" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/rollups.20vs.20testing.html#204210441">(Jul 17 2020 at 14:13)</a>:</h4>
<p>but different:)</p>



<a name="204210457"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/rollups%20vs%20testing/near/204210457" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/rollups.20vs.20testing.html#204210457">(Jul 17 2020 at 14:13)</a>:</h4>
<p>i.e., the same idea of "land a bunch of stuff fast, then go back and fill in the gaps"</p>



<a name="204210486"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/rollups%20vs%20testing/near/204210486" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/rollups.20vs.20testing.html#204210486">(Jul 17 2020 at 14:13)</a>:</h4>
<p>well -- in this case it'd probably be more like "fill in the gaps if needed"</p>



<a name="204210487"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/rollups%20vs%20testing/near/204210487" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/rollups.20vs.20testing.html#204210487">(Jul 17 2020 at 14:13)</a>:</h4>
<p>but it's probably better to ensure tests pass on the full set before landing, and have the "gaps we're filling in" just be binaries that aid in bisection</p>



<a name="204210514"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/rollups%20vs%20testing/near/204210514" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/rollups.20vs.20testing.html#204210514">(Jul 17 2020 at 14:13)</a>:</h4>
<p>well</p>



<a name="204210582"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/rollups%20vs%20testing/near/204210582" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/rollups.20vs.20testing.html#204210582">(Jul 17 2020 at 14:14)</a>:</h4>
<p>yeah I guess we could wait to do it until we find a regression from a rollup?</p>



<a name="204210590"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/rollups%20vs%20testing/near/204210590" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/rollups.20vs.20testing.html#204210590">(Jul 17 2020 at 14:14)</a>:</h4>
<p>right, yeah, that'd be the idea</p>



<a name="204210608"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/rollups%20vs%20testing/near/204210608" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/rollups.20vs.20testing.html#204210608">(Jul 17 2020 at 14:14)</a>:</h4>
<p>however, I'm also thinking of things like perf regressions</p>



<a name="204210622"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/rollups%20vs%20testing/near/204210622" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/rollups.20vs.20testing.html#204210622">(Jul 17 2020 at 14:14)</a>:</h4>
<p>which are harder to track down it seems</p>



<a name="204210657"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/rollups%20vs%20testing/near/204210657" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/rollups.20vs.20testing.html#204210657">(Jul 17 2020 at 14:14)</a>:</h4>
<p>I think we don't have time budget to build even just binaries for every PR in a rollup</p>



<a name="204210741"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/rollups%20vs%20testing/near/204210741" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/rollups.20vs.20testing.html#204210741">(Jul 17 2020 at 14:15)</a>:</h4>
<p>I think for perf regressions I'd rather have a bot that generates a revert PR for you for the "candidate" and then perf-tests that, which gives you a pretty good idea of the impact</p>



<a name="204210901"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/rollups%20vs%20testing/near/204210901" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/rollups.20vs.20testing.html#204210901">(Jul 17 2020 at 14:16)</a>:</h4>
<p>but yes I agree that looking into these things would be amazing</p>



<a name="204374426"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/rollups%20vs%20testing/near/204374426" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> njn <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/rollups.20vs.20testing.html#204374426">(Jul 19 2020 at 22:57)</a>:</h4>
<p>Don't tell Graydon you're considering this :)</p>



<a name="204381434"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/rollups%20vs%20testing/near/204381434" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/rollups.20vs.20testing.html#204381434">(Jul 20 2020 at 02:31)</a>:</h4>
<p>I'm frankly terrified. I've already complained about rollups (not made on the basis of small impactless PRs) making it hard to track down perf/functionality regressions. although some of the stuff I see in the discussion (extra binaries) seems sensible if done on top of what we have today</p>



<a name="204385181"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/rollups%20vs%20testing/near/204385181" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/rollups.20vs.20testing.html#204385181">(Jul 20 2020 at 04:13)</a>:</h4>
<p>Chrome does the equivalent of rollups for every merge, and then has some logic to guess the most likely failed commit that broke a merge and kick it out and try again. Very much a "computer time is cheaper than human time" solution.</p>



<a name="204408026"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/rollups%20vs%20testing/near/204408026" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/rollups.20vs.20testing.html#204408026">(Jul 20 2020 at 10:48)</a>:</h4>
<p>This reminds me, there are CI systems out there that do this as well, I remember being linked to one such but I don’t have a link to it handy anymore.</p>



<a name="204408074"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/rollups%20vs%20testing/near/204408074" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/rollups.20vs.20testing.html#204408074">(Jul 20 2020 at 10:49)</a>:</h4>
<p>basically it would test multiple potentially dependent commits in parallel and if there were failures it would retry the other PRs after booting the earlier failures… something along those lines.</p>



<a name="204408179"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/rollups%20vs%20testing/near/204408179" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/rollups.20vs.20testing.html#204408179">(Jul 20 2020 at 10:50)</a>:</h4>
<p>That wouldn’t work as well in our case as we already utilize pretty much all of the resources we have available to us to just cover all the different configurations.</p>



<a name="204408381"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/rollups%20vs%20testing/near/204408381" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/rollups.20vs.20testing.html#204408381">(Jul 20 2020 at 10:53)</a>:</h4>
<p><span class="user-mention" data-user-id="123586">@nagisa</span> was it maybe SubmitQueue <a href="https://blog.acolyer.org/2019/04/18/keeping-master-green-at-scale/">https://blog.acolyer.org/2019/04/18/keeping-master-green-at-scale/</a> ?</p>



<a name="204408430"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/rollups%20vs%20testing/near/204408430" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mati865 <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/rollups.20vs.20testing.html#204408430">(Jul 20 2020 at 10:54)</a>:</h4>
<p>Finding regressions caused by rollups isn't  cheap human time as well.</p>



<a name="204408533"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/rollups%20vs%20testing/near/204408533" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/rollups.20vs.20testing.html#204408533">(Jul 20 2020 at 10:55)</a>:</h4>
<p><span class="user-mention" data-user-id="116113">@lqd</span> doesn’t look like it. From what I remember it was an actual CI product website/landing page. Also had a video prominently on the front page or similar describing how their approach works.</p>



<a name="204408675"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/rollups%20vs%20testing/near/204408675" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/rollups.20vs.20testing.html#204408675">(Jul 20 2020 at 10:57)</a>:</h4>
<p>Though maybe the idea is the same, don’t have the time to dig into it now.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>